Enable the CacheLibWrapper class as a RocksDB Plugin#184
Enable the CacheLibWrapper class as a RocksDB Plugin#184mrambacher wants to merge 12 commits intofacebook:mainfrom
Conversation
facebook-github-bot
added
the
CLA Signed
| } | ||
| } | ||
|
|
||
| bool RocksCachelibWrapper::UpdateMaxWriteRateForDynamicRandom( |
There was a problem hiding this comment.
Can this not be supported anymore? Internally, we need this API.
There was a problem hiding this comment.
@anand1976 This is a feature that is internal to Facebook/Meta. The header file that is required for this is not part of the public API.
| defaultPool = | ||
| cache->addPool("default", cache->getCacheMemoryStats().cacheSize); | ||
|
|
||
| if (opts.fb303Stats) { |
There was a problem hiding this comment.
Is CacheAdmin for stats not supported?
There was a problem hiding this comment.
@anand1976 This is another piece that is internal to Facebook/Meta and is not in the public API.
| ``` | ||
| This will allow RocksDB to find and build the CacheLib plugin code. | ||
|
|
||
| Next, under the RocksDB build directory, instruct RocksDB to build and include the cachelib plugin: |
There was a problem hiding this comment.
I'm curious, how is the folly dependency handled? Should the user do this first - https://github.com/facebook/rocksdb/wiki/RocksDB-Contribution-Guide#build-rocksdb-with-folly-integration-enabled?
|
Is this the official open-source secondary cache implementation mentioned in the paper Disaggregating RocksDB: A Production Experience? |
|
What’s the status of this PR? This would be incredibly useful and something I think should get moved over the finish line. Is this something I can take over to give it the final push? |
5 participants
Took the existing code and got it to compile, build, and pass the tests as a RocksDB Plugin. Wrote a README with instructions on how to use and enable it.
In addition to verifying that the class could be created via the customizable_test, I built and executed the unit test (requires changes to the build as outlined in the README). I also verified via db_bench (command "./db_bench --secondary_cache_uri="id=RocksCachelibWrapper;size=256M;cachename=db_bench;filename=/tmp/db_bench_cache") that the secondary cache could be created.
I am not sure how to validate that the SecondaryCache is doing the right thing, but the test works without failures. A db_bench with readrandom appears to be slower with the SecondaryCache than without:
./db_bench --compression_type=none --num=1000000 --benchmarks=fillseq,readrandom
fillseq : 32.053 micros/op 31198 ops/sec 32.053 seconds 1000000 operations; 3.5 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
readrandom : 51.583 micros/op 19386 ops/sec 51.583 seconds 1000000 operations; 2.1 MB/s (1000000 of 1000000 found)
./db_bench --secondary_cache_uri="id=RocksCachelibWrapper;size=256M;cachename=db_bench;filename=/tmp/db_bench_cache" --compression_type=none --num=1000000 --benchmarks=fillseq,readrandom
fillseq : 32.136 micros/op 31118 ops/sec 32.136 seconds 1000000 operations; 3.4 MB/s
DB path: [/tmp/rocksdbtest-1000/dbbench]
readrandom : 67.078 micros/op 14907 ops/sec 67.078 seconds 1000000 operations; 1.6 MB/s (1000000 of 1000000 found)